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ABSTRACT 

Professional development and learning has a long history’ in seminar-like models, as well as in the more educator- 
personal delivery’ approaches. The question is whether an intentionally coordinated, integrated combination of the 
two PDL approaches will have best impacts for educators as quantified in improved student performance. 
Contrasts between baseline and Post-Program performance levels showed 19% gains in Reading and 24%> gains in 
Math, significantly beyond expectation. Analyses for Title 1 schools showed significant shrinkage of performance 
gaps with contrasted non-Title 1 schools. These gains outpaced those found for either PDL approach alone, 
indicating that educational leaders will be wise to undertake implementation of intentionally aligned and 
coordinated approaches combining PDL Seminars with online, on-demand PDL. 

Keywords: Professional Development; Professional Learning; Educational Benefits; Outcomes and Results; Student 
Advancement/Achievement 


BACKGROUND & INTRODUCTION 



rofessional development and professional learning (PDL) is a long-standing approach to providing 
value to teachers and educators (c.f. Avalos, 2011; Borko, 2004; Villegas-Reimers, 2003; Wayne, 
et.al, 2008; Wilson & Berne, 1999). The purpose of PDL has appropriately been to help educators 
feel more effective, refine and progress their skills and capabilities as educators, and have a greater impact on the 
students they have chosen to serve. Quantified proof of PDL impact is one challenge for educational leaders whose 
responsibilities include spending precious educational funds. Expenditures fail to be investments unless the higher 
aims of education are achieved. 


Most PDL offerings still lack any substantive or rigorous evidence of verified favorable educational impacts (c.f. 
Boyle, et.al, 2004; Garet, et.al, 2001). Marketing materials and testimonials alone are inadequate as a rationale base 
for the expenditure of limited funds whose purpose should be enhancing educator impacts on students - decision 
rationale must be based on more than participant satisfaction alone (c.f. Avalos, 2011; Borko, 2004; Villegas- 
Reimers, 2003; Wilson & Berne, 1999). The most rigorous research metrics for PDL evaluation should focus on 
teacher impact on student performance (c.f. Borko, 2004; Garet, et.al. 2001; Wayne, et.al. 2008). Educator PDL 
approaches with rigorous proof of favorable impacts on student gains should be preferred over those lacking such 
substantive cost justification. 

Educator PDL approaches have generally evolved in two directions, including PDL group seminars and PDL 
offerings more focused on personal participation which have recently matured into Internet-delivered PDL. But for 
research purposes, we choose to focus on the seminars and Internet-delivered approaches for our quasi-experimental 
undertakings, since each can be readily and rigorously studied through pre-versus-post or have-versus-have-not 
designs. 

A brief review of research for each PDL direction is appropriate and helpful, and raises crucial questions. 

Copyright by author(s); CC-BY 15 


The Clute Institute 



Contemporary Issues in Education Research - First Quarter 2016 


I plume 9, Number 1 


SEMINAR-BASED PROFESSIONAL DEVELOPMENT 

Beyond college-based training and experience, seminars have been a traditional basis for defining and delivering 
professional development for educators. Teachers customarily participate in conference-like settings and are 
exposed to concepts and methods of teaching designed to refine or endow a host of educator-relevant skills. While 
some seminar PDLs are conducted in calendar months when schools are not in session, many seminars require that 
teachers leave the classroom to participate, requiring replacement teachers or other student management approaches, 
all costing more funds, loss of classroom time with the designated day-to-day teacher, and turmoil (c.f. Abdal-Haqq, 
1996). 

Many studies have quantified impacts of PDL-focused seminars for educators (Borko, 2004; Koehler, Mishra & 
Yahya, 2007; Wayne, et.al. 2008; Zeichncr, 2003). Very few quantify any impact on student performance, 
traditionally focusing more on measuring participant satisfaction and testimonials of value. As a result, it appears 
that PDL choices made by educational leaders are dependent upon marketing-related concepts, reflections and 
promises, and not on any substantiated or rigorously quantified impact teachers had on their students. We consider 
that an unfortunately inadequate situation (c.f. Villegas-Reimers, 2003). 

ONLINE, ON-DEMAND PROFESSIONAL DEVELOPMENT 

With the advent and expansion of technology, approaches to PDL have evolved in the past 20-30 years toward 
greater use of options undertaken individually by educators, including video, computer-based options, and most 
recently Internet-delivered options (c.f. Schrum, 1995). The most advanced PDL offerings now leverage the 
ubiquity of the Internet to expand to online, on-demand capability offerings accessible whenever and wherever most 
convenient for educator enhancement (c.f. Dede, 2006; Schlager & Fusco 2003; Schrum, 1995; Wayne, et.al., 2008). 
Most of those advanced offerings are achieved through Internet-delivered video (c.f. Borko, et.al, 2008; Koc, Peker, 
& Osmanoglu, 2009). The cost effectiveness of Internet-delivered PDL in comparison to seminars has been part of 
its success and the expansion of commercial providers and offerings, resolving replacement-teachers expenses and 
other costs associated with participative seminar approaches (c.f. Abdal-Haqq, 1996; Garet, et.al. 2001; Zhang, et.al. 
2011 ). 

Further, recent rigorous studies have substantiated the impact of online, on-demand PDL approaches, at least for one 
of the market offerings (c.f. Shaha & Ellsworth, 2013, 2014a; Shaha, Glassett & Copas 2015a, 2015b; Shaha, 
Glassett & Ellsworth, 2015). In these studies, the percentage of students classified as Proficient and Advanced in 
Math and Reading increased 18% more year-over-year in online PDL schools then for their own respective (i.e. 
matched) non-participating districts. And teachers experienced those advantages after the first year of the online, 
on-demand PDL use. Similar study findings for the same Internet-delivered PDL were quantified for Title 1 
schools, which experienced 4.6% and 7.3% more students classified as Proficient and Advanced year-over-year for 
Reading and Math, respectively, versus the matched Title 1 control group which was either flat (0.1% more) or 
suffered performance losses (-5.9%) for the same year-over-year contrasts (Shaha & Ellsworth, 2014b; Shaha, et.al., 
2015). 


SUMMARY AND OBJECTIVES 

With the impact data so substantive and clear, PDL decisions can and should be driven by a focus on increasing the 
impact educators have on their students. Clearly the impact of the Internet-enabled online, on-demand PDL for 
educators is established, as well as the focus generating and foundation-setting impact of seminars is legitimate. 
However, with both seminars and online approaches shown to have favorable impacts in general, what educators 
need is a proven approach combining the best of both worlds (c.f. Shaha & Ellsworth, 2014a). Yet a review of the 
literature, however, did not lead to any research with findings regarding PDL offerings reflecting the coordinated, 
combined strengths of both approaches. 

The question yet to be answered is whether a combination of the two approaches can be shown with data to be 
effective, and potentially even more impactful than what either approach has achieved separately. For decision 
makers, can the cost-effectiveness, convenience, ready-access and proven impactfulness of Internet-delivered PDL 
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achieve even greater efficacy when integrated with an aligned PDL seminar? Further, would any potential benefits 
be generalizable to Title 1 school students? 


METHOD 

Combined data from nine previous studies of educational institutions were included in a meta-analysis of impact. 
Each study reflected inclusion categorizable as a uniform integrated PDL Program (hereafter, PDL Program), 
including both the same PDL Seminar throughout (i.e. Learning Framework®, Schools Improvement Network, Salt 
Lake City), integrated with the same online, on-demand, Internet-delivered PDL for teachers (PD 360 ®, School 
Improvement Network, Salt Lake City). Analyses only included schools for which the two PDL approaches were 
intentionally implemented as an integrated, aligned, and coordinated undertaking. All data analyzed were included 
at the collective school level in order to ensure anonymity for teachers and students. 

The sample included 52 total schools in seven districts within five of the states, with 39 elementary schools (75%), 
three middle schools (6%), and 10 high schools (19%). Schools collectively represented geographic diversity from 
west to east, and north to south, as well as urban to rural, reflecting an estimated 1,400+ teacher and 26,000+ 
students. The sample represented a nearly average American ethnic mix (see Table 1; Aud, et al, 2010; Lee, 2002). 
The designation of “Title 1” for any school was preserved from a previous study for analysis of that group of 
schools separately (c.f. Shaha, et.al., 2015), which in this data set included 14 schools in four districts within four of 
the five states in the original data set. 


Table 1. Cumulative Ethnic Mix in the Sample 


Sample 

White 

50.1% 

Hispanic 

19.1% 

Black - African American 

17.1% 

Asian / Pac Islander 

8.2% 

Native American 

1.1% 

Multi Ethnicity 

4.4% 

Total 

100% 


The study design reflected a quasi-experimental approach (Cook & Campbell, 1979) wherein performance of 
naturalistically occurring classroom groups (i.e. school aggregates) were contrasted for three successive school years 
in a 3-year model. All data included in the meta-analysis were shifted such that years aligned to ensure analogous 
starting and ending parameters. Thus while specific calendar years for data varied between schools due to 
respective implementation schedules, all occurred between 2008 and 2013, with the same 3-year model applied 
uniformly. Within the 3-successive-year design: 

• Pre-Year 1 data reflected mean student reading and mathematics performance school-wide for each 
school two full academic years before implementation. 

• Pre-Year 2 data reflected performance data for the academic year closing immediately prior to the 
aligned, coordinated PDL implementation. Comparison of Pre-year 1 versus 2 quantified probable 
change rates in student performance without either the online, on-demand PDL or the Seminar. 

• Post data reflected the year-end performance for the academic year of the combined PDL intervention. 

Student performance was the metric for program impact evaluation. To adjust for different student performance 
instruments (i.e. standardized tests) between states, student performance for all analyses was defined as the sum of 
the percentages of students rated either proficient or advanced on their respective standardized student performance 
tests, for reading and mathematics (thus entitled despite varying labels). This approach also ensured generalizability 
(Shaha, Glassett & Copas, 2015; Shaha, Glassett & Ellsworth, 2015). Importantly, no changes in testing 
instrumentation occurred within any included states or districts for the three consecutive years of data. Student 
performance data were gathered from publically available web sources for the three consecutive school years. 

Year-to-year percent change - gain or loss from start to end for each year - was used as the main measure for 
statistical analysis and interpretation, thus standardizing the basis of analysis regardless of either the calendar-year 
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or absolute start or end points for any element within the sample. For analyses referring to Title 1 schools, the year- 
over-year change in percent of student classified as proficient or advanced was maintained and computed for an 
additional 14 Title 1 schools within the same states. However, the more relevant analyses focused on evaluating any 
change in the gap in student performance when contrasted with non-Title 1 schools (c.f. Aud, et al, 2010; Lee, 
2002 ). 

Percent change for reading and mathematics were computed as net change divided by performance for the earlier of 
the two years contrasted: e.g. (Year 2 - Year 1)/ Year 1. Effect size was then computed and reported as the percent 
change Pre-Year 2 to Post, divided by the percent change for the previous periods, as well as by Cohen's f (Cohen, 
1992). Significance for percent change was determined by the t-Test for proportions (Rice, 2006). All analyses 
were conducted using SPSS version 17.0 or higher (PASW Statistics, SPSS, 2009, with SAS for confirmatory 
purposes when results were close to p<0.05). Minimum level of statistical significance was determined a priori at 
p<0.05. 


RESULTS 

Contrasting Pre-Year 1 and Pre-Year 2 reflected favorable changes of 1.9% and 1.7% year-over-year for Reading 
and Math, respectively (see Table 2 and Figure 1). These growth rates reflect the pre-implementation rates, and thus 
provide a baseline from which to assess the program impact (Pre-Year 2 versus Post). Neither growth rate was 
statistically significant for any given school when analyzed alone, hence the designation of “not statistically 
significant” indicated within the table and illustrated in the figure. When the full collective sample size is reflected 
in cumulative analyses, then the percentages were significantly different from zero. 


Table 2. Year-over-Year changes in Student Performance with and without the Integrated Program 



Baselines 

Program Impacts 


Pre-Year 1 

Pre-Year 2 

Year-over-Year 
Performance Change 

Normal 

Growth 

Post 

Program 

Year-over-Year 
Performance Change 

Program-enabled 
Growth Rate 

Reading 

48.2% 

49.1% 

0.9% + 

1.9% + 

58.9% 

9.8% ** 

19.96% ** 

Math 

45.9% 

46.7% 

0.8% + 

1.7% + 

58.1% 

11.4% ** 

24.41% ** 


t - Not statistically significant 

** - Statistically significant at p<0.001 


Figure 1. Year-over-Year changes in percent Advanced and Proficient Student Performance with the Integrated Program 
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In contrast, results for the Post Program year revealed highly favorable gains and significant Program impacts from 
the Pre-Year 2 baselines for Reading and Math, alike, far outpacing the baseline rates for Year 1 to Year 2 (see 
Table 2 and Figure 1). All told, the year-over-year impacts on student performance attributable to the Program were 
19.96% and 24.41% for Reading and Math, respectively. 
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Restated, those figures indicated that for Reading, for example, one student of every five who was previously 
performing at lower classification levels experienced a promotion to Proficient or Advanced performance levels 
beyond the 1.9% change at baseline (p<0.001). Math showed even more dramatic impact, wherein nearly one 
student of every four students previously performing at lower classification levels now performed higher and at a 
Proficient or Advanced performance levels versus the former baseline change of 1.7% (p<0.001). 

IMPACT ON ECONOMICALLY DISADVANTAGED STUDENTS 

Additional analyses were conducted for the additional 14 schools classified as Title 1 for evaluating the PDL 
Program impact on the population best representing economically disadvantaged (c.f. Aud, et al, 2010; Lee, 2002; 
Shaha, Glassett & Copas, 2015b). To best represent the impact of the PDL Program’s impact for Title 1 schools, the 
amount of performance gap between Title 1 and non-Title 1 schools was quantified and analyzed beyond year-over- 
year change alone (c.f. Aud, et al, 2010; Lee, 2002). Non-Title 1 schools were selected from the 52 in the previous 
data reflecting best possible matches geographically. 

The gap in percentage of student performing at Proficient and Advanced levels was not significantly reduced 
between the two pre-PDL Program years. While a slight reduction in the gap was achieved for the 14 Title 1 
schools collectively, the gap’s closing was very nearly 0.0% for both Reading and Math (see Table 2). However, 
once the PDL Program was in place - both online, on-demand PDL and the participative PDL Seminar - the 
performance gap fell substantially and significantly (see Table 3 and Figure 2). For Reading, the gap for percentage 
of students classified as Proficient and Advanced, for Title 1 versus non-Title 1 schools, was reduced by more the 
35% for Reading from the last Pre-Year baseline to the end of the first Post Program year (p<0.001): The gap 
experienced a net reduction of 8.3% for the percentage of students at Proficient and Advanced levels of 
achievement. For Math the impact was even greater, as the Proficient and Advanced gap fell by more the 40% in 
the first Post Program year (p<0.001), a net gap reduction of over 11% for the percentage of students at Proficient 
and Advanced levels of achievement. 


Table 3. Performance Gap for Title 1 versus Non-Title 1 Schools in Percent of Students Proficient and Advanced 



Baselines 

Program Impacts 


Pre-Year 1 

Pre-Year 2 

Year-over-Year 
Gap Change 

Normal Change 
Rate 

Post Program 

Year-over-Year 
Gap Change 

Program-enabled 
Change Rate 

Reading 

23.8% 

23.6% 

-0.2% t 

-0.8% t 

15.3% 

-8.3% ** 

-35.17% ** 

Math 

27.7% 

27.6% 

-0.1% t 

-0.4% t 

16.3% 

-11.3% ** 

-40.94% ** 


t - Not statistically significant 

** - Statistically significant at p<0.001 


Figure 2. Falling gap for percent students Proficient and Advanced for Title 1 versus non-Title 1 schools 
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INTERPRETATIONS AND CONCLUSIONS 

Results indicate that educational leaders and their organizations would substantively benefit from the execution of 
such integrated approaches to both PDL Seminars and online, on-demand PDL for active educators. The 
combination of online, on-demand professional development and learning with a coordinated, integrated PDL 
Seminar resulted in impacts on student performance significantly more than prior to the implementation, or to what 
could be expected for either approach separately. 

Specifically, for example, the nearly 20% improvement in Proficient and Advanced student-count achieved through 
integrated approach for Reading performance represents student gains 10.3% higher than the 18.1% year-over-year 
growth shown in analogous studies of online, on-demand PDL approach alone (c.f. Shaha & Ellsworth, 2013; Shaha, 
Glassett & Copas, 2015a ). For math, the over-24% gain in Proficient-Advanced student-count was more than 29% 
higher than the 18.9% advantage documented for the studies of the online PDL alone. 

Data further showed that the effect was equally remarkable for Title 1 schools with student populations often 
described as economically disadvantaged. Closing performance gaps represents evidence of the long-sought 
capability to establish a more equally impactful approach to education, regardless of the populations of students 
being served. 

A host of recommendations emerge from these analyses, some from the data and some from the common-sense 
experiences of careers as educators. First, it is strongly recommended that educators heed evidence and pursue the 
most effective PDL approaches proven with research and data. From previous studies it is clear that, in order to 
maximize their impact on student performance, educators should undertake well-initiated and actively pursued 
online, on-demand PDL for their active teachers and educators. Now an additional enhancement has been defined 
and assessed: it is substantiated that incorporation of an integrated, aligned PDL Seminar in concert with the online, 
On-demand PDL should enable even more substantively favorable impacts for teachers, as measured in student 
performance. 

Second, it is recommended that decision-makers be conservative in their anticipations and expectations, and in their 
goals and promises. Analyses of averages, as found in this and most rigorous studies, describe general patterns that 
mask both the extraordinarily better than average in the sample, as well as the unfortunately lower-than-average 
within the same data. Investing in PDL approaches that benefit from the integration of both seminars and Internet- 
delivered PDL is a no-brainer, as the saying goes, and should be quickly and aggressively pursued. However, 
setting goals that reflect percentages of change rather than massive gross changes in student performance will 
generally lead to more realistic expectations and more celebrated accomplishments thereafter. It is unclear whether 
all schools in all districts and states, and all teacher cadres and student populations, can or will achieve the levels of 
accomplishment found in these data. That said, however, there is no reason to anticipate lesser impacts. 
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